[FlyDSL] Add gfx950 FHMoE tuner in fhmoe_tune.py by a-canadasruiz · Pull Request #5751 · ROCm/aiter

a-canadasruiz · 2026-09-22T08:55:12Z

Summary

Add a tuner for the DeepSeek-V4 FHMoE rows in aiter/configs/tuned_fhmoe.csv on gfx950: csrc/ck_gemm_moe_2stages_codegen/fhmoe_tune.py.
Those rows already serve through fused_moe, but nothing in the repo could produce or re-check them.

Motivation

DSv4 FHMoE runs routed MXFP4 experts plus one FP8 shared expert in a single launch (E=385, topk 7, shared expert 384, GateMode.INTERLEAVE). There was no tuner for its rows in the repo: no way to retune a shape, add a new one, or check with --run_config that the shipped rows still match torch.

Since #6050, the same CSV also holds the HY4 rows, tuned by op_tests/tuners/tune_fhmoe.py. This tuner only touches rows with MXFP4 weights (DSv4) and skips the FP8-weight HY4 rows. The HY4 tuner can't cover DSv4 as it is: its shape, FP8 weights, kernel list and reference are fixed in the script, and it only retunes rows that already exist, on one GPU. This tuner reads the shape from untuned_fhmoe.csv, can add new shapes, runs on several GPUs through mp_tuner and has --run_config. One FHMoE tuner for both models would be a follow-up.

Changes

The commits follow this list.

Tuner. FhmoeTuner in its own module, fhmoe_tune.py, next to gemm_moe_tune.py, plus the catalogue untuned_fhmoe.csv (12 serving buckets, 1 to 2048 tokens).
- It subclasses TunerCommon and imports a few helpers from gemm_moe_tune.py (weight quant, the torch stage references, cosine_diff_compare). gemm_moe_tune.py is not changed.
- Each candidate is one FlyDSL stage1 x stage2 pair with a block_m. Stage 1 comes in fused and non-fused quant (_fp8 suffix). That is about 5632 pairs per token. The kernel list follows the row's activation and weight dtypes, as in FmoeTuner.
- Each pair is timed through _fused_moe_impl with a one-row temp CSV. The public fused_moe() cannot take that file, because DSv4 serving needs every padded-M tier in the config.
- Pairs that miss the torch reference are dropped. The fastest of the rest is written under get_padded_M(token).
- Only DSv4 rows are rewritten. Tokens that are not serving buckets are rejected. Rows pinned to another gfx/cu_num are skipped.
- Weights are quantized on the CPU, so the GPU worker only runs fused_moe.
CPU tests for FhmoeTuner (test_fhmoe_dsv4_tuner.py), run in the level 0+1 job.
Tuning Tests coverage as family fhmoe_dsv4: CSV validation, --run_config, a small tune pipeline, and a manual replay test.
op_tune.sh: a new fhmoe job, skipped on anything but gfx950. It does not pass --all, so it is a no-op until a new shape is added to the catalogue.
Reference that quantizes like the kernel. Rows with MXFP8 activations (DSv4, INTERLEAVE) quantize them before stage 1 and between the stages. The reference now does the same, in two variants (stage-1 result rounded to bf16 first, or quantized from fp32 like a fused-quant stage 1), and the candidate is judged against the closer one. --run_config tries the fp32 variant only when the first one misses. Serving code (fused_moe, aiter/fhmoe.py) is not changed. The shipped tuned_fhmoe.csv is not changed.

Testing

PR Self-Review

Reviewed revision

8b80abe

Behavior and intent

Adds a tuner path only: a new script, fhmoe_tune.py. Before this PR the DSv4 rows of tuned_fhmoe.csv could not be retuned or checked with --run_config. gemm_moe_tune.py is not changed. Serving and the shipped CSV are unchanged.

Validation and unverified scope

On MI355X (gfx950), at this revision, all passed:

test_fhmoe_dsv4_tuner, test_csv_validation (CPU, 51 tests)
test_fhmoe_replay (3 tests)
test_tune_pipeline ... test_fhmoe_dsv4_mp1
test_run_config ... test_fhmoe_dsv4: 12 shipped DSv4 rows, no ERROR or MISMATCH

Not verified:

Rows with fp4 activations still use an unquantized reference. The tuner warns. No such rows ship today.
Full sweeps only for token 1.
tuning-tests.yaml runs on schedule or workflow_dispatch only, so this PR's CI does not run it.

PR-specific question

No new permissions, secrets or triggers. op_tune.sh gets one job entry and a gfx950 check. tuning-tests.yaml adds one module to the level 0+1 command. Both workflows run only on schedule or workflow_dispatch, so fork PR code or text cannot reach them.

Changed op tests

test_fhmoe_dsv4_tuner.py: CPU tests for FhmoeTuner, including that a rewrite keeps the HY4 rows. Nothing else covers it.
test_fhmoe_replay.py: replays the tuner output through the public fused_moe. Manual, gfx950.
test_csv_validation.py: no duplicate shapes, catalogue tokens are serving buckets.
test_run_config.py: adds the fhmoe_dsv4 family.
test_tune_pipeline.py: small --mp 1 tune for fhmoe_dsv4.
README.md: documents the new tests and family.
test_fhmoe.py: __main__ now runs pytest.main. Before, it exited 0 without running any test.

github-actions · 2026-09-22T08:56:02Z

🏷️ CI Guide

Runs automatically on every PR:

✅ Pre-checks (submodule verification, code formatting)
✅ Aiter op tests (gfx942 + gfx950)
✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
`ci:gfx1250-ffm-triton`	Run the five-shard gfx1250 FFM Triton test suite
`ci:triton-300x`	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
`multigpu`	Aiter multi-GPU tests on the 8-GPU runner
`ci:sglang`	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
`ci:atom`	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
`ci:atom_full`	ATOM accuracy suite for PR and main models from ATOM `models_accuracy.json`
`ci:vllm`	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
`ci:all`	All standard extended tests (excludes `ci:atom_full`)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5751 --add-label <label>

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to opt this PR out.

aiter-bot · 2026-10-29T14:54:57Z

NOT READY FOR HUMAN REVIEW

PR Self-Review

Revision 8b80abecd1dd · mode shadow

Author self-review accepted. Readiness report:

Author answers

PASS — current-SHA answers are present and structurally specific

Show the author's self-review answers

Behavior and intent

Adds a tuner path only: a new script, `fhmoe_tune.py`. Before this PR the DSv4 rows of `tuned_fhmoe.csv` could not be retuned or checked with `--run_config`. `gemm_moe_tune.py` is not changed. Serving and the shipped CSV are unchanged.

Validation and unverified scope

On MI355X (gfx950), at this revision, all passed:

- `test_fhmoe_dsv4_tuner`, `test_csv_validation` (CPU, 51 tests)
- `test_fhmoe_replay` (3 tests)
- `test_tune_pipeline ... test_fhmoe_dsv4_mp1`
- `test_run_config ... test_fhmoe_dsv4`: 12 shipped DSv4 rows, no ERROR or MISMATCH

Not verified:

- Rows with fp4 activations still use an unquantized reference. The tuner warns. No such rows ship today.
- Full sweeps only for token 1.
- `tuning-tests.yaml` runs on schedule or `workflow_dispatch` only, so this PR's CI does not run it.

PR-specific question

No new permissions, secrets or triggers. `op_tune.sh` gets one job entry and a gfx950 check. `tuning-tests.yaml` adds one module to the level 0+1 command. Both workflows run only on `schedule` or `workflow_dispatch`, so fork PR code or text cannot reach them.

Changed op tests

- `test_fhmoe_dsv4_tuner.py`: CPU tests for `FhmoeTuner`, including that a rewrite keeps the HY4 rows. Nothing else covers it.
- `test_fhmoe_replay.py`: replays the tuner output through the public `fused_moe`. Manual, gfx950.
- `test_csv_validation.py`: no duplicate shapes, catalogue tokens are serving buckets.
- `test_run_config.py`: adds the `fhmoe_dsv4` family.
- `test_tune_pipeline.py`: small `--mp 1` tune for `fhmoe_dsv4`.
- `README.md`: documents the new tests and family.
- `test_fhmoe.py`: `__main__` now runs `pytest.main`. Before, it exited 0 without running any test.

PR state

WAIT — pull request is still a draft

Copilot findings

PASS — no active Copilot finding lacks a substantive author reply
ADVISORY — Copilot did not review the current head SHA

Code and op-test hygiene

PASS — no committed artifact or conflict marker detected

Required CI

PASS — Black: passed
PASS — Ruff: passed
PASS — Repository dependency: passed
PASS — Aiter Test Gate: passed

_{Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.}

Add FhmoeTuner in its own module, fhmoe_tune.py, for the DeepSeek-V4 INTERLEAVE FHMoE path on gfx950 (routed MXFP4 experts plus one FP8 shared expert). It subclasses TunerCommon and imports the few helpers it needs from gemm_moe_tune.py, which is left unchanged. - untuned_fhmoe.csv lists the serving buckets, 1 to 2048 tokens. - Each task times one FlyDSL stage1 x stage2 pair through fused_moe, fused and non-fused stage1 quant, with --warmup/--iters. - Weights are quantized on the CPU (DSv4 MXFP4 quant peaks at several GiB), so the worker only times fused_moe on the GPU. - Candidates are checked against a torch reference before timing. - Rows are published under the padded token and only the tuner's own rows of tuned_fhmoe.csv are rewritten. - Tokens that are not serving buckets are rejected, and rows pinned to another gfx/cu_num are skipped.

Cover token buckets, kernel pairs, post_process, the CSV rewrite and the gfx/cu_num guard without a GPU, and run them in the level 0+1 job.

- test_csv_validation: no duplicate shapes, and catalogue tokens are serving buckets. - test_run_config: fhmoe_dsv4 family. - test_tune_pipeline: small mp=1 tune on gfx950. - test_fhmoe_replay: replay the tuner output through the public fused_moe in a fresh process (manual, gfx950).

Add an fhmoe job and skip it on anything but gfx950, so Operators Tuning on gfx942 does not fail.

… the kernel MXFP8 activations quantize between and inside the stages. Compare each candidate with the closest of two torch references that quantize the same way (bf16-rounded and fp32), and have run_config retry with the fp32 one when the first misses. Rows with fp4 activations still use the unquantized reference; warn when a row is checked that way.

fxmarty-amd · 2026-11-03T11:38:35Z

RElated: https://github.com/ROCm/aiter/blob/main/op_tests/tuners/tune_fhmoe.py ?

a-canadasruiz force-pushed the users/acanadas/fhmoe-tuner branch 3 times, most recently from 25a6fb6 to 1dfc0b3 Compare September 23, 2026 11:37

amd-bartgips mentioned this pull request Sep 23, 2026

[MHA] Route MHA forward through a tuned backend row #5762

Open

a-canadasruiz changed the title ~~[FlyDSL] Add gfx950 FHMoE producer on gemm_moe_tune.py~~ [FlyDSL] Add gfx950 FHMoE tuner on gemm_moe_tune.py Sep 24, 2026

amd-bartgips mentioned this pull request Sep 25, 2026

[CK] [CI] [JIT] [Tuning] Per-candidate faults, selection proof and auto-select in --compare #5846

Open

a-canadasruiz force-pushed the users/acanadas/fhmoe-tuner branch from c12acdc to bdac6e9 Compare September 25, 2026 11:43

amd-bartgips mentioned this pull request Sep 25, 2026

[Tuning] A family-agnostic race for large candidate fields (draft, not for review yet) #5849

Draft

a-canadasruiz force-pushed the users/acanadas/fhmoe-tuner branch from 6d19303 to f1321b1 Compare October 13, 2026 10:38

amd-bartgips mentioned this pull request Oct 15, 2026

[JIT] Fix an indefinite JIT build hang when a lock holder is a zombie #6579

Draft

a-canadasruiz force-pushed the users/acanadas/fhmoe-tuner branch 2 times, most recently from 685936f to 3e81963 Compare October 22, 2026 15:32

a-canadasruiz force-pushed the users/acanadas/fhmoe-tuner branch 2 times, most recently from eb191ba to 1febc43 Compare October 29, 2026 11:18

ROCm deleted a comment from aiter-bot Oct 29, 2026

a-canadasruiz force-pushed the users/acanadas/fhmoe-tuner branch from 1febc43 to d766370 Compare October 29, 2026 13:58

ROCm deleted a comment from aiter-bot Oct 29, 2026

a-canadasruiz added 5 commits October 30, 2026 16:40

[FlyDSL] Add CPU tests for FhmoeTuner

bf2a57a

Cover token buckets, kernel pairs, post_process, the CSV rewrite and the gfx/cu_num guard without a GPU, and run them in the level 0+1 job.

[FlyDSL] Run the FHMoE tuner from op_tune.sh

3d3d54d

Add an fhmoe job and skip it on anything but gfx950, so Operators Tuning on gfx942 does not fail.

a-canadasruiz force-pushed the users/acanadas/fhmoe-tuner branch from d766370 to 8b80abe Compare October 30, 2026 17:14

a-canadasruiz changed the title ~~[FlyDSL] Add gfx950 FHMoE tuner on gemm_moe_tune.py~~ [FlyDSL] Add gfx950 FHMoE tuner in fhmoe_tune.py Oct 30, 2026

fxmarty-amd mentioned this pull request Nov 3, 2026

[ROCm] Heterogeneous fused shared experts for DeepSeek-V4.1 (native-FP8 shared slot, prefill-shaped batches) vllm-project/vllm#63180

Open

6 tasks done

Navigation Menu

Repository navigation

[FlyDSL] Add gfx950 FHMoE tuner in fhmoe_tune.py - #5751

[FlyDSL] Add gfx950 FHMoE tuner in fhmoe_tune.py#5751
a-canadasruiz wants to merge 5 commits into
ROCm:mainfrom
a-canadasruiz:users/acanadas/fhmoe-tuner

a-canadasruiz commented Sep 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

aiter-bot commented Oct 29, 2026 •

edited

Loading

Behavior and intent

Validation and unverified scope

PR-specific question

Changed op tests

Uh oh!

fxmarty-amd commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

a-canadasruiz commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

Testing

PR Self-Review

Reviewed revision

Behavior and intent

Validation and unverified scope

PR-specific question

Changed op tests

Uh oh!

github-actions Bot commented Sep 22, 2026

🏷️ CI Guide

Uh oh!

aiter-bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

NOT READY FOR HUMAN REVIEW

PR Self-Review

Author answers

Behavior and intent

Validation and unverified scope

PR-specific question

Changed op tests

PR state

Copilot findings

Code and op-test hygiene

Required CI

Uh oh!

fxmarty-amd commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

a-canadasruiz commented Sep 22, 2026 •

edited

Loading

aiter-bot commented Oct 29, 2026 •

edited

Loading